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DETAILED ACTION 

Response to Remarks 

1 . The Office Action has been issued in response to amendment filed January 5, 
2007. Claims 1-4-12, 14-24, and 26-32 are pending. Applicant's arguments have 
been carefully and respectfully considered in light of the instant amendment, and 
are not persuasive. Accordingly, this action has been made FINAL. 

Specification Objection 
The specification has been amended to correct the typographical error. 
Therefore, the objection has been removed. 
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112 Rejections 

Claims 17 and 18 have been amended to provide antecedent basis for all claim 
terms. Therefore, the rejection has been removed. 

101 Rejections 

Claims 31 and 32 have been amended to include the phrase "computer readable 
medium," thereby making both claims statutory. Therefore, the rejection has been 
removed. 

Response to 103 Rejection Arguments 

In response to Applicant arguments page on page 13 [p][002] that Lou fails to 
disclose at least the feature of that the training set of pixels to provide using a user input 
which exhibits sample color arrangement associated with the first and second visual 
texture and that Lou teaches away from classifying images on the basis of texture, the 
Examiner disagrees with Applicant. Lou clearly teaches a method for defining a 
boundary (differentiate true sky regions from other similarly color and textured subject 
matters; column 4, lines 33-34) separating a first region (e.g. non-sky-colored pixels, 
column 5, line 41) and a second region (e.g. sky colored pixels; column 5, line 41) of a 
digital image (200, Fig.2), the digital image including one or more color arrangements 
characteristic of a first visual texture of (see Fig 12 A, where the first region includes 
water and a boat which has various colors) of the first region and one or more color 



Application/Control Number: 10/693,295 Page 4 

Art Unit: 2624 

arrangements characteristic of a second visual texture (sky color which includes non- 
uniform blue and almost white colors; column 6, lines 59-65) of the second region and 
receiving a user input providing training sets of pixel exhibiting sample color 
arrangements associated with the first and the second and first visual texture (note that 
various color sample of sky and negative examples that are neither blue, sky or water 
were provided as training sets; see column 8, lines 19-35). Although, Lou does not 
specifically mention receiving a user input, it would have been obvious that a user or 
operator would have to provided the inputs for the classifier. Furthermore, Lou clearly 
teaches classifying images on the basis of texture (see column 2, lines 59-62). 

In response to Applicant arguments page on page 13 [p][003] that Lou fail to 
disclose that a neural network or a learning machine is trained to classify learning 
machine input sets based upon the training set, where each learning machine set inputs 
includes a pixel of interest and neighboring pixel, in which it determined which pixels of 
the digital image satisfy criteria for classification as associated with the first region and 
the second region by inputting learning machine inputs (including the pixel of interest 
and neighboring pixels) and outputting an indication of a region to which each pixels of 
interest belong, the Examiner disagrees with Applicant. Lou clearly teaches training a 
color classifier (note that a neural network is used in the training process, see column 8, 
lines 29-31) using color samples of sky and also note that negative examples that are 
neither blue, sky or water were also used during training (see column 8, lines 19-35). 
Lou also teaches that if the overall belief value of candidate region is above a certain 
threshold, the regions is declared a sky (see column 6, lines 18-19) and if gradient value 
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is above predetermined threshold, the region is considered a non-sky regions (see 
column 5, lines 52-61). 

In response to Applicant's argument on page 15, [p][001] that Norton is not seem 
to remedy the deficiencies of Lou, the Examiner disagree. Norton teaches a method of 
masking an object (column 1 , line 59) including identifying pixels (a pixel is identified if it 
in a certain range, column 16, lines 45-50) and decontaminating the identified pixels to 
define the boundary between the first and second regions (column 17, lines 1-4) and 
therefore cures the deficiencies of Lou. Applicant has also argued that Norton fails to 
disclose: i) a user input is received providing training sets of pixel exhibiting sample 
color arrangements associated with the first and the second and first visual texture, ii) a 
neural network or a learning machine is trained to classify learning inputs sets based 
upon training set, each learning machine input set including a pixel of interest and 
neighboring pixels and being derived from pixels in the digital image and iii) it is 
determined which pixels of the image satisfy criteria for classification as associated with 
the first region or second region, by inputting learning machine input sets and outputting 
an indication of a region which each of the pixels of interest belong. However, Norton 
was not relied upon for the rejection of the above-mentioned limitations. 

In response to Applicant's argument on page 15, [p][002] that Boskovitz does 
not remedy the deficiencies of Lou or Norton, the Examiner, disagrees. Boskovitz 
teaches a method for multilevel image segmentation (page 247, [p][3], lines 1-2) that 
includes wherein: neighboring pixels represent is one of a three-by-three (see Fig 6b) 
square of pixels and a five-by-five square of pixels (see Fig 6c) and therefore remedy 
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the deficiencies of Lou and Norton. Applicant has also argued that Norton fails to 
disclose: i) a user input is received providing training sets of pixel exhibiting sample 
color arrangements associated with the first and the second and first visual texture, ii) a 
neural network or a learning machine is trained to classify learning inputs sets based 
upon training set, each learning machine input set including a pixel of interest and 
neighboring pixels and being derived from pixels in the digital image and iii) it is 
determined which pixels of the image satisfy criteria for classification as associated with 
the first region or second region, by inputting learning machine input sets and outputting 
an indication of a region which each of the pixels of interest belong. However, 
Boskovitz was not relied upon for the rejection of the above-mentioned limitations 

Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 1.03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-3, 5-7, 13-16, 20, 25 and 31 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Luo et al (US Patent No.: 6,504,951 ) in view of Norton 
(US Patent No.: 5,912,994). 



Application/Control Number: 10/693,295 Page 7 

Art Unit: 2624 

As to independent claim 1 , Luo discloses a method for defining a boundary 
(differentiate true sky regions from other similarly color and textured subject matters; 
column 4, lines 33-34) separating a first region (e.g. non-sky-colored pixels, column 5, 
line 41) and a second region (e.g. sky colored pixels; column 5, line 41) of a digital 
image (200, Fig. 2), the digital image including one or more color arrangements 
characteristic of a first visual texture of (see Fig 12 A, where the first region includes 
water and a boat which has various colors) of the first region and one or more color 
arrangements characteristic of a second visual texture (sky color which includes non- 
uniform blue and almost white colors; column 6, lines 59-65) of the second region, the 
method comprising: receiving a user input providing training sets of pixel exhibiting 
sample color arrangements associated with the first and the second and first visual 
texture (e.g. various color sample of sky and negative examples that are neither blue, 
sky or water, see column 8, lines 19-35); training a learning machine to classify learning 
inputs sets based upon training set, each learning machine input set including a pixel of 
interest and neighboring pixels and being derived from pixels in the digital image (note 
that the color classifier was trained using color samples of sky and negative examples 
that are neither blue, sky or water, see column 8, lines 19-35); Lou teaches determining 
using the trained learning machine (color classifier, column 8, line 47) based on one or 
more of the color arrangements, which pixels of the image satisfy criteria (if gradient 
value is above predetermined threshold, the region is considered a non-sky regions; 
column 5, lines 52-61) for classification as associated with the first region or second 
region, by inputting learning machine input sets and outputting an indication of a region 
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which each of the pixels of interest belong (if the overall belief value of candidate region 
is above a certain threshold, the regions is declared a sky column 6, lines 18-19). Luo 
further teaches identifying pixels of the image that are determined not to satisfy the 
criteria for classification as being associated either with the first region or the second 
region (marginal pixels; column 10, line 53) and decontaminating the identified pixels to 
define the boundary between the first and second regions by separating pixels of the 
digital image into pixels associated with the first region, the second, or the boundary 
(note that each pixel is classified either as sky, non-sky or marginal, see column 10, 
lines 33-55). 

However, Luo does not teach identifying pixels and decontaminating the 
identified pixels to define the boundary between the first and second regions. Norton 
teaches a method of masking an object (column 1 , line 59) including identifying pixels (a 
pixel is identified if it in a certain range, column 16, lines 45-50) and decontaminating 
the identified pixels to define the boundary between the first and second regions 
(column 17, lines 1-4). Therefore it would have been obvious to one of ordinary skill in 
the art at the time the invention was made to have added the method of masking an 
object to the method of defining a boundary of Luo for detecting a boundary of a 
substantially color-homogeneous region of a frame of a digitized picture stock (column 
1, lines 61-65) with "increased efficiency" (column 1, line 54). Also, the process of 
Norton allows for better segmentation of the various regions of the image and better 
identification of ambiguous pixels. 
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As to independent claim 31 , this claim differs from claim 1 only in that claim 31 is 
a computer program product whereas, claim 1 is method and the limitations computer- 
readable medium comprising instructions and a processor are additively recited in the 
preamble. Lou clearly teaches a computer program product comprising instructions 
(column 3, line 6). Note the discussion above, Norton teaches computer-readable 
medium (memory, column 3, line 24) comprising instructions operable and a processor 
(12, see Fig 1). 

As to claim 5, Luo teaches a method wherein the learning machine is a neural 
network (column 5, lines 4-5) 

As to claim 6, Luo teaches a method wherein the indication represents a 
probability of the pixel being associated with the first region and a probability of the pixel 
of interest being associated with the second region (see column 8, lines 47-50, where a 
pixel classifier is trained to output a belief value between 0 and 1 a pixel, 1 indicating a 
highly likely to be blue sky and 0 indicating not very likely to be blue sky). 

As to claim 7, Lou teaches a method wherein the indication is a floating point 
(number between a lower number (e.g. 0; column 8, line 45) and an upper number (e.g. 
1; column 8, line 45) the lower number indicating a one-hundred percent probability of 
the pixel of interest being associated with the second region (e.g. 0 indicating the 
likelihood a pixel as non-sky; column 8, line 45) and the upper number indicating a one- 
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hundred percent probability of the pixel of interest being associated with the first region 
(e.g. 1 indicating the likelihood a pixel ideal blue sky; column 8, line 45). 

As to claim 13, Luo teaches a method of training the learning machine to classify 
pixels (column 8, lines 20-21). 

As to claim 14, Lou teaches a method wherein training includes selecting, based 
on user input, a training set of pixels used to train the neural network (column 8, lines 
30-35). 

As to claim 15, note the discussion above, Norton teaches the method wherein 
the training set of pixels selected includes pixels located within a particular range of the 
boundary (see Fig 6, step 208, where a range can be adjusted). 

As to claim 16, note the discussion above, Norton teaches a method wherein the 
particular range is 20 pixels from either side of the boundary. In the spec (page 19, 
[p][22], line 10), Applicant states that the particular range can in the range of 2 to 100 
pixels. Consequently, there is no significant advantage or disclosed criticality in 
choosing the particular range to be 20 pixels, therefore it would have been obvious that 
the particular range can be 20 pixels. 

As to claim 20, note the discussion above, Norton teaches constructing from the 
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identified pixels a boundary mask that indicates which pixels of the digital image are the 
identified pixels (column 17, lines 16-21). 

4. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Luo et al 
(US Patent No.: 6,504,951) in view of Norton (US Patent No.: 5,912,994) further 
in view of Wang et al (Pub No.: US 2003/0007683). 

As to claim 4, Lou in view of Norton does not teaches a method wherein the 
learning machine is a support vector machine. Wang teaches a method of separating 
text from drawings (page 1 , [p][0005], lines 1-2) that includes using a support vector 
machine as a learning machine (page 1, [p][0006], lines 4). Therefore it would have 
been obvious to one of ordinary skill in the art at the time the invention was made to 
have substituted the color classifier of Luo with the support vector machine of Wang for 
classifying a stroke according to the slope curvature vector as either "text" or "unknown" 
(page 1, [p][0006], lines 8-10), because each reference region is for segmenting out one 
region from another and the use of a support vector machine is well known to segment 
regions of an image. 

5. Claims 8-12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Luo 
et al (US Patent No.: 6,504,951) in view of Norton (US Patent No.: 5,912,994) 
further in view of Littmann et al (NPL Document titled "Apative Color 
Segmentation - A comparison of Neural and Statistical Methods). 
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As to claim 8, Lou teaches wherein the lower number is -1 and the upper number 
is 1 (column 8, line 45). However Both Lou and Norton do not teach wherein the lower 
number is -1 . Littmann teaches a method of comparing learning classifiers (page 175, 
[p][2], line 4-6) including wherein the lower number is -1 (page 176, [p][3], line 7). 
Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have added the method of comparing classifier of Littmann to the 
method of defining a boundary of Luo as modified by Norton for classifying pixels in an 
image as foreground (e.g. hand, page 175, [p][4], line 6) and background (page 175, 
[p][4], line 6), and because this simply gives a commonly used value to represent one 
of the conditions. 

As to claim 9, note the discussion above, Littmann teaches a method of 
converting the floating point number to an integer between a first integer and a second 
integer, the first integer indicating a one-hundred percent probability of the pixel of 
interest being associated with the second region, and the second integer indicating a 
one-hundred percent probability of the pixel of interest being associated with the first 
region (see page 179, [p][7], lines 7- 8, where the output the classifier are scale to gray 
values). 

As to claim 10, note the discussion above, Littmann teaches a method wherein 
the first integer is 0 and the second integer is 256 (see page 179, [p][7, lines 8-9 where 
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the above mentioned gray values are in the range of [0, 256]). Also, these are the 
standard values for gray value image data. 

As to claim 1 1 , Luo teaches the criteria for classification as associated with the 
first region includes having an integer that exceeds a first threshold; and the criteria for 
classification as associated with the second region includes having an integer that is 
less than a second threshold (see column 9, lines 25-32, where the value of each pixel 
is proportional to its belief value and a global threshold is used to create a binary map, 1 
is considered a potential sky pixel and 0 is considered a non-sky pixel). 

As to claim 12, Lou teaches the method of claim 1 1 , wherein: the first threshold 
is 170 and the second threshold is 85. There is no real significant advantage for 
choosing the first threshold to be 170 and second threshold to be 85, therefore it 
obvious that the first and second threshold can be 1 70 and 85 respectively. 

6. Claim 17-19, 24, 26-30 and 32 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Luo et al (US Patent No.: 6,504,951) in view of Norton (US 
Patent No.: 5,912,994) further in view of Boskovitz et al (NPL Document titled 
"An Adaptive Neuro-Fuzzy System for Automatic Image Segmentation and Edge 
Detection"). 
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As to claim 17, the combination of Lou and Norton do not teach the method of 
claim, wherein: the neighboring pixels represent is one of a three-by-three square of 
pixels, a five-by-five square of pixels, and a seven-by-seven square of pixels. 

Boskovitz teaches a method for multilevel image segmentation (page 247, [p][3], 
lines 1-2) that includes wherein: neighboring pixels represent is one of a three-by-three 
(see Fig 6b) square of pixels and a five-by-five square of pixels (see Fig 6c). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to have added the method for multilevel image segmentation of 
Boskovitz to the method of defining a boundary of Luo as modified by Norton for 
segmentation of intensity image (page 261, [p][1], line 1) without "require any human 
expert intervention, nor any a priori information about the input image" (page 261, [p][1], 
lines 20-23). Also, the use of 3x3 and 5x5 windows of pixels to process image data is 
extremely conventional and allows a large image to be processed in smaller, easier to 
process blocks. 

As to claim 18, note the discussion above, Boskovitz teaches the method 
wherein the pixel being considered is located at a center of the neighborhood of pixels 
(see Fig 6a). 



As to claim 19, note the discussion above, Boskovitz teaches the learning 
machine is a neural network (see Fig. 5); the neural network includes hidden nodes and 
gating nodes (see Fig. 5); and a gating node is associated with a corresponding hidden 
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node, the gating node being configured to determine, based on a location of the pixel of 
interest being considered, a contribution the corresponding hidden node makes to an 
output of the neural network (page 263, [p][1]). 

As to independent claim 24, note the discussion of claims 1 and 19 above. 

As to claim 26, note the discussion of claims 17 and 18 above. 

As to claim 27, note the discussion above, Boskovitz teaches the method of 
training the gating node to determine, based on the location of the pixel of interest being 
considered, a contribution the hidden node makes to the output of the neural network 
(page 253, [p]4]). 

As to claim 28, note the discussion above, Boskovitz teaches the method of 
training the hidden nodes to classify pixels of the digital image as either associated with 
the first region or associated with the second region, wherein the training of the hidden 
nodes occurs during the training of the gating nodes (page 253, [p]4]). 

As to claim 29, note the discussion above, Boskovitz teaches the method further 
comprising: receiving input information specifying the location of the pixel being 
considered using input nodes; and providing the input information to the gating node 
(see Fig 5). 
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As to claim 30, note the discussion above, Boskovitz teaches the method futher 
compromising: receiving input information specifying the color arrangement of the 
corresponding neighborhood of pixels via input nodes; and providing the input 
information to the corresponding hidden node (see Fig 5). 

As to independent 32, all the limitation are discussed above except: wherein the 
neural network includes a gating note associated with a corresponding hidden node, the 
gating node being configured to determine, based on a location of the pixel of interest, a 
contribution the corresponding hidden node makes to the output of the neural network. 
(Note the discussion of claim 24 above). 

7. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over Luo et al 
(US Patent No.: 6,504,951) in view of Norton (US Patent No.: 5,912,994) further 
in view Simon et al (Pub. No.: US 2004/0170337). 

As to claim 21 , Both Luo and Norton do not teach a wherein decontaminating 
produces an opacity mask, the method further comprising: constructing from the 
identified pixels a probability mask; and combining the opacity mask and the probability 
mask. Simon teaches. a method for enhancing an appearance of a face located in a 
digital image (page 2, [p][12], lines 3-4). Simon also teaches wherein decontaminating 
produces an opacity mask (generate alpha channel mask; page 6, [p][59], line 16-17), 
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the method further comprising: constructing from the identified pixels a probability mask 
(page 6-7 [p][61], line 8); and combining the opacity mask and the probability mask 
(page 8, [p][72], lines 11-12). Therefore it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to have added the method for 
enhancing an appearance an image of Simon to the method of defining a boundary of 
Luo as modified by Norton to "enable the retouching of portrait without requiring skilled 
operator intervention to make and supervise the retouching correction" (page 2, [p][13], 
lines 3-5). 

As to claim 22, note the discussion above, Simmon teaches the method of 
combining the opacity mask and the probability mask includes multiplying the opacity 
mask with the probability mask (see page 7, [p][64], lines 4-5, where combining 
probability maps includes arithmetic multiplication). 

8. Claim 23 is rejected under 35 U.S.C. 103(a) as being unpatentable over Luo et al 
(US Patent No.: 6,504,951) in view of Norton (US Patent No.: 5,912,994) further 
in view Mao (Pub No.: 2003/0063797). 

As to claim 23, Lou teaches the method wherein the first region is a foreground 
(e.g. nonsky pixels such as the water and the boat in Fig. 12 A) of the digital image and 
the second region is a background (e.g. sky in figure 12 A) of the digital image. Note 
the discussion above, Norton teaches decontaminating includes excluding from the 
identified pixels a pixel that has no foreground colors (adjust border range, column 16, 
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line 41-43). However, Both Lou and Norton do not teach changing colors of a pixel that 
includes both foreground and background colors so that the changed identified pixels 
include only foreground colors. Mao teaches a selection tool for defining a border area 
(page 1 , [p][0009] that includes teach changing colors of a pixel that includes both 
foreground and background colors so that the changed identified pixels include only 
foreground colors (change the color of a pixel in a target pixel map to foreground color) 
(page 4, [p][0048]). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to added the selection tool of Mao to the method of 
defining a boundary of Luo as modified by Norton to simplify "selection of complicated 
objects by providing a wide selection brush" (page 4, [p][0052], lines 2-4) and because 
each reference is for segmenting region of an image. 

Conclusion 

9. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
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the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Inquires 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Andrae S. Allison whose telephone number is (571) 
270-1052. The examiner can normally be reached on Monday-Friday, 8:00 am - 5:00 
pm, EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph Mancuso can be reached on (571) 272-7695. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from the 
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